Proceedings of the ICML / UAI / COLT Workshop on Abstraction in Reinforcement Learning

Bayesian Reinforcement Learning (BRL) provides an optimal solution to on-line learning while acting, but it is computationally intractable for all but the simplest problems: at each decision time, an agent should weigh all possible courses of action by beliefs about future outcomes constructed over long time horizons. To improve tractability, previous research has focused on sparsely sampling possible courses of action that are most relevant to computing value; however, sampling alone does not scale well to larger environments. In this paper, we investigate whether an abstraction called projects— parts of the transition dynamics that bias the look ahead to areas of the environment that are promising—can scale up BRL to larger environments. We modify a sparse sampler to incorporate projects. We test our algorithm on standard problems that require effective exploration–exploitation balance and show that learning can be significantly sped up compared to a simpler BRL and the classic Q-learning.